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ABSTRACT 

Al 

Since  Griffith  and  Stewart  firstly  proposed  as  successive  linear  programming 
method  for  solving  general  nonlinear  programming  problems*  such  methods  have 
been  widely  used  in  practice  because  of  their  ease  of  implementation  and  their 
ability  to  deal  with  large  scale  problems.  However*  neither  the  original 
version,  nor  a  more  recent  one  contain  convergence  proofs  possibly  because  of 
non-robustness  of  their  algorithms.  Using  exact  penalty  functions  and  Levenberg- 
Marquardtlike  steps,  an  Improved  algorithm  has  been  recently  devised.  In  this 
paper,  we  give  this  modified  SLP  method  a  theoretical  analysis  and  convergence 
proof,  and  thereby  provide  a  sound  basis  for  it.  - 

N 
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A  Successive  Linear  Programming  Method  and  Its  Convergence 
on  Nonlinear  Problems 

by 

J.  Zhang 


11  Introduction 


In  this  paper  we  discuss  the  following  type  of  general  nonlinear  programming 


problem 


min  f (Xj )  +  pxg 

s.t.  9i(x1)  +  q^x2  ■  b.  ,  i  * 

g^(xj)  +  q^x2  <  b.  ,  i  *  k+l,...,h 


(1.1) 


(ai  *A2 )  n )  <  c 


where  XjCR0,  x^R1",  the  vectors  p,  q^€Rm,  Aj  and  A?  are  respectively 
1  x  n  and  1  x  m  matrices,  cCR1,  f  and  gi  (i  =  l,...,h)  are  continuously 
differentiable  functions  in  Rn.  For  simplicity,  we  write  the  scalar  product 
of  vectors  x  and  y  as  xy.  When  there  is  no  need  to  differentiate  between  the 

nonlinear  part  and  the  linear  part  of  the  problem  functions,  we  write 


F(x)  A  f(Xj )  +  px2 

Gi(x)  A  g^Xj)  +  q^^x2  -  bj  ,  i  =  l,...,h 
Here  x  *  (x^  ,x2) <  Rn+m. 

In  1961,  Griffith  and  Stewart  [1]  proposed  a  successive  linear  programming 
method  for  the  above  problem  (1.1).  In  their  method,  linear  approximations  for 
all  nonlinear  functions  f  and  g^  were  made  and  then  the  resulting  linear  program¬ 
ming  problem  was  solved  in  a  bounded  region.  Because  of  its  ease  of  implementation 
and  its  ability  to  deal  with  large  scale  problems,  this  method  has  been  widely 
used  in  various  organizations.  However,  although  this  method  has  worked  in 
many  examples,  its  convergence  has  not  been  established  in  general  cases. 


Since  its  appearance,  some  modified  and  Improved  versions  have  appeared, 
such  as  [2],  [3],  For  example,  in  [3]  using  penalty  functions,  another  SLP 
method  has  been  presented.  Computational  results  for  it  compare  favorably  with 
the  well-known  Generalized  Reduced  Gradient  Method.  However,  the  algorithm  of 
[3]  is  rather  complicated  and  Its  convergence  was  only  proved  (in  an  unpublished 
dissertation)  for  linearly  constrained  problems.  Using  the  basic  idea  of  [3], 
Lasdon,  Kim  and  this  author  Improved  the  SLP  algorithm  further  in  [4].  The  new 
SLP  method  has  worked  efficiently  in  extensive  sets  of  test  examples.  In  this 
paper  we  give  the  method  a  theoretical  analysis  and  convergence  proof  and  thereby 
provide  a  sound  basis  for  it. 

In  the  next  section,  §2,  we  discuss  the  exact  penalty  function  of  problem 
(1.1).  In  §3  we  introduce  the  Improves  SLP  algorithm,  and  In  §4  we  solve  the 
problem  of  convergence  of  this  method. 

§2  Some  Relative  Results  on  Exact  Penalty  Function 
For  problem  (1.1),  we  construct  the  exact  penalty  function 

k  h 

P(z)  A  F(x)  +  E  u>.  | G . (x)  |  -*■  I  w.  max  (0,G.(x))  (2.1) 

i=l  1  1  i=k+l  1  1 

where  w  *  (ujj . u>h)  and  i  are  suitable  positive  numbers  (i  *  l,...,h). 

Now  we  consider  the  problem 

min  P(x)  (2.2) 

Ax  <  c 


There  are  many  papers  devoted  to  expounding  the  close  relation  between  the 
optimal  solution  of  the  exact  penalty  function  problem  and  that  of  the  original 
problem.  According  to  the  results  of  [5],  and  considering  the  linear  inequality 
constraints,  which  still  remain  in  (2.2),  we  can  easily  find  the  relation 
between  problem  (2.2)  and  (1.1). 


ttMSMKKKh? 


mmmm^s&ciK&u 


bV 

bv 


Set 

ft  a  {x  :  G^(x)  a  0  ,  i  *  l,...,k  ; 

G..(x)  <  0  ,1  =  k+1,. .. ,h  ; 
Ax  <  c> 


and 

ft'  *  {x  :  Ax  <  c}  . 

We  can  obtain  the  following: 

Proposition  2.1  If  there  exists  a  w  >  0  such  that  for  all  u>  >  w,  7  is  a  local 
minimum  point  of  the  penalty  function  P(x,w)  in  N(7)nft'»  which  contains  at 
least  one  feasible  point  of  original  problem  (here,  N(7)  is  an  open  neighborhood 
of  x),  then  xGft  and  is  a  local  optimal  solution  of  problem  (1.1).  Especially, 
if  for  some  <*>,  the  local  optimal  solution  7  is  a  feasible  point  of  (1.1),  then 
x  must  be  a  local  optimal  solution  of  the  original  problem. 

For  the  original  problem  (1.1),  if  we  write  the  last  constraint  as  xeft',  then 
under  any  common  K-T  type  constraint  qualification  (for  instance,  the  qualifica¬ 
tions  of  Mangasarian-Fromovitz  or  Kuhn-Tucker  in  [6]),  we  can  state  the  necessary 
condition  for  an  extremum  point  7  as  follows: 

There  are  u  =  (u^ ,. . .  .u^) 
v  =  ^uk+r*'”uh^ 

satisfying 

7.  >  G  ,  u..G.(7)  =  0  ,  i  =  k+1 . h  (2.3) 


and  for  VsCT(ft'.x),  we  always  have 

h 

(vf(7)  +  E  7.  vG,(7))  s  >  0 
i=l  1  1 


9 
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(2.4) 
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where  T(fl',x)  is  the  tangent  cone  of  linear  constraint  set  ft'  at  x: 


T(fl',x)  =  {s  :  a^  <  0  ,  1€I'(x)} 


(2.5) 


I ' (x)  #  {i  :  i €  (1,2,... ,1}  and  a^x  =  c^} 


(2.6) 


ai  is  the  i-th  row  vector  of  matrix  A. 

In  fact,  (2.3)  and  (2.4)  form  an  optimality  criteria  of  the  minimum 
principle  type,  see  [6],  [7].  Of  course,  we  can  also  rewrite  (2.4)  as 

l>f(x)  +  £  u.  ^(x)  +  £  u.  ^(x)]  s  >  0 
i=l  1  1  i€l(x) 


where 


I ( x)  A  {1  :  i«{k+l,... ,h}  and  G^(x)  «  0}. 


Furthermore,  it  is  easy  to  modify  the  standard  Fiacco-McCormlck  second  order 
sufficient  condition  [8]  as  follows: 

If  the  feasible  point  x  of  problem  (1.1)  satisfies  the  necessary  conditions 
(2.3)  and  (2.4),  and  for  any  s  j4  0  with  the  following  properties: 


VF^jOs  <  0 

VG. (x)s  ~  0  ,  i  =  1,...  ,k 
VGi(x)s<0  i€l(7) 


s  e  T(ft' ,x) 


The  following  strict  inequality  always  holds: 


S[v2f  +  Eu^G.Vs  >  o  t 
1=1  1  1  x 


(2.7) 


then  x  is  a  strict  local  optimal  solution  of  the  original  problem  (1.1). 
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Consulting  the  proof  of  lemma  4.5  and  theorem  4.6  of  [5],  it  is  not  difficult 
to  obtain: 


Proposition  2.2  If  the  feasible  point  x  of  problem  (1.1)  satisfies  second  order 
sufficient  conditions  (2.3),  (2.4)  and  (2.7),  and  w  *  (u>j». . .  ,w^)  has  the  property 

min  {w. }  >  max  { |tT ^  | } 

i“l . h  i-1,. . . ,h 


then  x  must  be  a  strict  local  minimum  solution  of  exact  penalty  function  P(x,w) 
in  Q' . 

From  the  above,  we  know  that  except  in  some  particular  cases  (for  example, 
see  [9]),  usually  by  solving  problem  (2.2),  we  can  get  the  optimal  solution  of 
the  original  problem  (1.1). 

Now  we  turn  to  the  optimality  criteria  for  problem  (2.2).  Obviously,  P(x) 
is  a  nonsmooth  function.  However,  since  F(x)  and  G,.(x)  are  all  continuously 
differentiable,  P(x)  is  a  locally  Lipschitz  continuous  function  and  so,  according 
to  [10],  we  know  that  along  every  direction  s,  function  P(x)  has  generalized 
directional  derivative 


P°(x;s)  A  lim  sup  p(x'+6s)  ~  p(x') 


x  -*-x 
6  4-0 


(2.8) 


and  by  virtue  of  it,  Clarke  defined  the  generalized  gradient  of  P(x): 


3P(x)  A  {£  :  Vv«Rn+tn  ,  P°(x;v)  >  v*;}  (2.9) 

and  furthermore,  we  know  that  3P(x)  is  a  compact  convex  set  of  Rn+m  and 


P°(x;s)  -  max  cs  (2.10) 

C<9P(x) 

Because  the  constraints  in  problem  (2.2)  are  all  linear,  obviously  the  problem 
satisfies  common  constraint  qualifications  for  nonsmooth  extremum  problems  and 
therefore,  according  to  [11],  [12],  [13],  we  have 
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Proposition  2.3  The  necessary  condition  for  x  being  a  local  optimal  solution 
of  extremum  problem  (2.2)  is  that  for  i  =  1,2,. . .  ,1  and  i«  I'(x)  (see  (2.6)), 
there  exist  A..  >  0  such  that 

0  (  3P(x)  +  . 

i«r  (x)1  1 

Or,  i f  we  denote 

D(x)  A  3P(x)  +  Z  A. a.  (2:il) 

iU'tx)1  1 

Ai  >  0 

ft*  A  {x  :  0  e  D(x)} 

and  call  ft*  the  set  of  stationary  points  of  non-differentiable  programming  (2.2), 
then  Proposition  2.3  implies  that 

x  is  a  local  optimal  solution  of  (2.2)  *=■>  x«ft*. 

In  the  last  part  of  this  section,  we  are  going  to  discuss  the  structure 
of  the  generalized  gradient  3P(x).  First,  we  have 

proposition  2.4  If  </>(x,u^)  is  a  finite  family  of  functions,  in  which  x  <  Rn+m, 

(k)  (k)  (kl 

u'  y«R,  k  =  l,...,s;  and  for  every  u'  ,  <?(x,u'  )  is  a  continuously  differentiabl 

function  of  x,  then  the  function 

tp(x)  *  max  <p(x,u^) 
k-1 ,  •  • « »s 

is  a  locally  Lipschitz  continuous  function;  and  for  every  direction  v,  there 
is  a  directional  derivative  (x ; v )  under  the  usual  meaning,  which  equals 
t|>°(x;v);  and  furthermore, 

3^(x)  =  co  (Vx<p(x,u^)  :  u^«M(x)} 


(2.12) 


*•«*»*; 


M(x)  *  {u'  '  :  <p(x,uv  ')  =  ij;(x)> 


and  co(-)  denotes  the  convex  hull  of  set  (*)• 

In  fact,  this  proposition  is  a  particular  case  of  [14],  Ch.  Ill,  wherein 
Pshenichnyi  has  some  general  results  about  quasi -differentiable  functionals 
involving  extensive  arguments.  Here  we  give  a  straightforward  proof  for  our 


Proof 

Using  the  locally  Lipschitz  property  of  differentiable  function  v,  it  is  not 
difficult  to  see  that  0(x)  is  also  locally  Lipschitz  continuous. 

Now,  for  every  u^«  M(x), 

V«.(*,u(k))v  =  llm  *(«  +  «»  »i(k’)  •  ?(»,u(k)) 

6+0  6 


lim  inf 
6  +  0 


(since  <p(x  +  6v,u^)  <  ^(x  +  6v 


lim  sup 
6  +  0 


tp°(x;v) 


lim  sup 
x'-*-  x 
6  +  0 


[x1  +  6v)  -  ip (x 1 ! 
6 


max  <?(x'  +  6v,u'K' )  - 


max  <p(x’,u^) 


6 
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(when  x1  nears  x,  6  nears  0,  we  have 
H(x‘ )  c  M(x)  and  M(x'  +  6v)  c  M(x)  ) 


<  11.  sup  {  »(»'+sv,u(k)}  -?(»■, u(k)) , 

S';S  u<k).M(x)  6 

=  lim  sup  max  {V<p(x' ,u^)v  +  o(l)> 

;';S  u^m(x) 

=  max  V<p(x,u^)v  (2.13) 

u(k}<M(x) 


max  wv 

w  t  co  {Vv>(x,u^)  :  u^«M(x)} 

Obviously,  at  the  very  beginning  of  the  series  of  expressions  above,  we  can 
select  the  particular  u^,  which  makes  the  expression  V<P(x,u^)v  in  (2.13) 
reach  its  maximum,  and  it  means  that  all  these  inequalities  must  be  held  as 
strict  equality.  So,  there  exists  ip'(x.v)  and 

lim  Mil  +  5v)  ~ 

6+0  6 

=  <J>'(x;v) 

=  i^°(x;v) 

=  max  wv 
w«co  {V<p(x,u^)  :  u^«M(x)} 


(2.14) 


On  the  other  hand,  by  virtue  of  (2.10), 


tp°  ( x  ;v )  =  max  *;v 
C«3<i>(x) 


(2.15) 


Because  both  sets  appeared  respectively  in  (2.14)  and  (2.15)  are  closed  convex 
sets,  so  according  to  Corollary  13.1.1  of  [15], 

3iJj(x)  =  co  (V<P(x,u^)  :  u^«M(x)} 

From  the  above  result,  now  it  is  easy  to  express  the  structure  of  9P(x). 

Proposition  2.5  For  every  positive  vector  u>,  the  generalized  gradient  set  of 
function  P(x)  defined  by  (2.1)  is 


9P(x)  =  (VF(x)  +  Z  u.o).  VG.(x)} 
i=l  1  1  1 


(2.16) 


and  for  i«l,. . . ,k. 


sgn  G.  (x) 

Ui  =  | [-1,1] 


if  G.j (x)  f  0 
if  G.(x)  =  0 


(2.17) 


for  i=k+l ,. . . ,h. 


[1  +  sgn  Gi(x)]/2  if  G^x)  t  0 


[0,1] 


if  Gi(x)  =  0 


(2.18) 


Proof 


Since  | G .  (x) |  =  max  {G..(x)-1  ,  Gi  (x)*-l> 


max  (G..(x),0)  =  max  (G^. (x)  - 1  ,  G..(x)*0} 


k  h 

1  u).  |G. (x)  |  +  Z  w.  max  (G.(x),0) 
i=l  1  1  i=k+l  1  1 


3 
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h 

s  max  {  £  oj .6. (x)u.) 
u«U  i=l  1  1  1 

Uj  =  ±1  when  i*l,...,k; 

where  U  A  {u  =  (u, ,. . .  ,u.  ,u.  . . u.)  :  } 

1  K  K  1  n  ui  a  0  or  1  when  1=k+l,...,h 


We  set 

h 

<?(x,u)  A  F(x)  +  E  ui.G.(x)u. 

i=l  1  1  1 


then  P(x)  =  max  <p(x,u) 
u«U 

and  by  virtue  of  Proposition  2.4,  we  have 

8P(x)  =  co  {V^(x,u)  :  u«M(x)} 

h  u.  are  determined 

=  co  {VF(x)  +  _E  u.u.VGi(x)  :  by  (2>17j  gnd  (2.18)} 

and  which  is  just  the  same  as  (2.16). 

Incidently,  Coleman  and  Conn  has  obtained  nearly  the  same  result,  see  [9], 
Theorem  1  and  its  Corollary  1,  but  in  their  proof,  they  assumed  that  vectors 
(VG..(x)  |  G..(x)  =  0,  i=l,2,...,h}  are  linearly  independent,  which  is  redundant 
according  to  the  proof  given  here. 

As  a  matter  of  fact,  according  to  Propositions  2.3  and  2.5,  it  is  obvious 
that  there  exists  another  relation  between  the  original  problem  (1.1)  and  the 
penalty  problem  (2.2). 

Proposition  2.6  If  "x  is  a  Kuhn-Tucker  point  of  problem  (1.1)  with  the  corresponding 
multipliers  (i=l,...,h)  and  I..  (i=l,...,l),  and  the  coefficients  u>..  of  the 
penalty  function  P  satisfy  the  condition  that 


11 


0).  >  |uf  I  ,  1-1,. ...h 

then  x  must  be  a  stationary  point  of  penalty  problem  (2.2). 


Proof 

By  K-T  conditions,  we  have 
h 

0  =  VF(x)  +  Z  u.VG.(x)  +  Z  X.a. 

1-1  1  1  i«I'(x)  1  1 


(u^  >  0  and  u^fx)  *  0  for  i»k+l,...,h; 

L  >  0  for  1«I'(x)) 
k 

GVF(x)  +  Z  as.  {u.VG.(x)  :  u.«[-l,l]} 
i=l  1  1  1  1 

♦  i «.  woo :  vt-1^  * if  si<!>  ■  n 

i=k+l  1  l1  1  u-  ®  0  if  G.(x)  <  Oj 


+ 


z  _ 

i«r(x) 


=  9P(x)  +  Z  X.a. 

i«r(x)  1  1 


Here,  the  last  equation  is  because  of  expressions  (2.17),  (2.18)  and  the  fact 
that  7  is  a  feasible  point  of  the  original  problem  (1.1). 

Now,  by  virtue  of  Propositon  2.3,  we  know  that 
0  « D(7) 

which  means  7  is  a  stationary  point  of  (2.2). 


§3  Successive  Linear  Programming  Algorithm 
Now  we  present  the  modified  SLP  algorithm  of  [4]  which  we  shall  employ  in  our 
proof.  Taking  a  linear  approximation  for  penalty  function  P(x),  that  means  for 
7  *  (Tj.Xg)  and  d  =  (dj,d2),  where  d^R0,  d2<Rm,  we  approximately  have 


A  (F(x)  +  7F(x)d)  +  r  u>.  |G. (x)  +  VG4Cx)d| 

H  1  1  1 

h 

♦  E  Ui  max  (0,G.(x)  +  VG. (x)d)  (3.1) 

1-k+l  1  1  1 

In  fact,  because  the  second  parts  of  F  and  G^  themselves  are  linear  functions, 

only  the  approximation  for  functions  f  and  Is  needed. 

The  problem  (2.2)  has  now  been  changed  approximately  to  a  linear  programming 
problem: 

min  R(x,d) 

s.t.  Id,!  <  a 

1  00 

Id-1  <  M  (3.2) 

Z  00 

Afx  +  d)  <  c 

Here,  the  reason  for  restricting  the  norm  of  vector  is  clear,  because  it  is 
only  when  we  do  so  that  the  linear  approximation  can  be  reasonable,  and  furthermore, 
the  value  of  a  should  be  adjusted  according  to  how  better  the  linear  approximation 
was  performed.  Positive  number  M  can  be  taken  as  any  large  amount.  From  the 
approximate  point  of  view,  it  is  unnecessary  to  restrict  the  norm  of  vector  dg i 
however,  in  order  to  guarantee  that  the  subproblem  (3.2)  must  have  a  finite 
optimal  solution,  we  need  to  restrict  d  in  a  bounded  area. 

Using  norm  can  make  (3.2)  easily  become  a  linear  programming  problem, 
and  for  simplicity,  from  now  on  we  just  use  II  •  II  instead  of  sign  II  •  11^.  Now 
we  state  the  whole  algorithm  as  follows: 


Set  k  k+1  ,  and  return  to  2°. 


»>-CM%<%WKi (*u 


««*■«*  i 


The  main  part  of  this  algorithm  Is  successively  solving  linear  programming 
(3.3).  We  can  rewrite  the  objective  function  R(x^,d)  as 

Vf(x,M)  d.  +  pd,  +  E  u).(st  +  ST)  +  Z  w.st  , 

l  l  z  iBl  1  1  1  1-k+1  i  1 

and  at  the  same  time  supplement  a  set  of  constraint  conditions: 

'g1(*l(k))  +  vg1(x1(k))<l1  +  q(1)(x2(k)  +  i2)  -  bf  •  s|  - 

■j  s|  >  o  ,  s;  >  0 

l  1  *  1.2, . . . ,h 


Now  the  subproblems  In  the  algorithm  become  ordinary  LP  problems. 

We  use  the  ratio  of  AP^  and  AR^  to  assess  whether  the  last  linearization 
Is  a  good  approximation  or  not,  and  then  to  decide  the  maximum  step  length  for 
next  iteration.  This  is  a  common  technique  in  Levenberg-Marquardt  type  methods 
for  nonlinear  least  square  problems  (see  [16]  and  references  therein). 

§4  Convergence 

In  this  section,  we  are  going  to  prove  the  convergence  of  this  modified 
SLP  algorithm.  The  main  idea  of  our  proof,  especially  the  logic  order  of  the 
following  Lemma  4.3,  Lemma  4.4  and  Theorem  4.5,  to  a  large  extent,  come  from 
Zhu  Meifang  and  author's  paper  [16],  in  which  we  gave  a  nearly  uni  ."arm  proof 
to  several  versions  of  Levenberg-Marquardt  algorithm  for  nonlinear  least  square 
problems.  We  find  that  the  way  of  dealing  with  the  convergence  of  restricted 
step  algorithms  used  in  [16]  is  also  very  useful  in  the  following  argument  and 
the  only  difference  is  that  now  we  have  to  notice  the  nondi fferentiable  nature 
of  the  present  problem. 

Theorem  4,1  The  AR^)  defined  in  above  algorithm  (k*l,2,...)  are  always 
non-negative,  and  if  AR^  *  0,  the  corresponding  x^  is  a  stationary  point  of 


fee 


programming  (2.2),  i.e.,  xv,w«  fl*,  and  It  is  the  case  of  finite  convergence  of 


Proof 


Obviously  d  ■  0  is  a  feasible  solution  of  problem  (3.3),  hence 


R(x*k*,0)  >  R(x(k\d^) 


(4.1) 


By  definition  of  function  R,  R(x^,0)  *  P(x^)  and  so,  (4.1)  Implies 
AR(k^  =  P(x(k^)  -  R(x*k*,d^)  >  0 


(kl 

Furthermore,  AR  -  0  if  and  only  if  d  ■  0  is  an  optimal  solution  of  problem 
(3.3).  Because  the  constraints  appeared  in  (3.3)  are  all  linear  inequalities, 
they  meet  usual  constraint  qualifications  and  the  first  two  of  them  are  obviously 
inactive.  So  according  to  the  Proposition  2.3,  it  is  certain  that  there  exist 
Aj  >  0,  1«r(x^),  such  that 

0  « 3dR(x(k^ ,d)  |d=0  +  1 €l?^x(k) jAiai  (4-2> 


where  a^  is  the  i-th  row  vector  of  A. 


Using  Proposition  2.4  and  Proposition  2.5,  but  instead  of  x  with  d  and  taking 
values  at  d  =  0,  we  get 


9dR(x(k),d)  |da0  =  VF(x(k))  +  I^w.VG.U^) 


and  where,  for  i  *  l,...,k. 


ui  ‘  [-i.i] 


sgn  G((x*k>)  if  G,(x^k,j  i>  0 


otherwise 


and  for  i  *  k+1 ,. . . ,h. 


ui  *  lco.n 


[1  +  sgn  G.,(xtk))]/2 


if  G^x^)  f  0 
otherwise 


The  above  result  Indicates  that 


3dR(xU,,d)  |d,Q  -  3P(x(k)) 


and  hence  from  (4.2)  and  (4.3)  we  know 


x*k><  n*. 


(4.3) 


In  order  to  prove  the  convergence  when  (x'KJ)  Is  an  Infinite  sequence,  we 
need  several  lemmas  at  first. 


Lemma  4.2  For  any  x,  we  have 


P(x  +  d)  =  R(x,d)  +  o( fldj  U ) 


(4.4) 


and  If  the  nonlinear  part  Xj  of  variable  x  Is  restricted  In  a  bounded  closed 
set  ACRn,  then  the  o(Hdjll)  Is  Independent  of  x  on  set  AxRm. 


Proof 

From  the  calculus,  we  know  that  If  function  f  Is  continuously  differentiable 
In  an  open  set  containing  A,  then  for  any  Xj<A,  we  have 

f  ( x  j  +  dj)  =  f  (xx )  +  VfUjJdj  +  o(  ld1 1 ) 

and  the  remainder  o ( !l d ^ II )  of  it  is  independent  of  Xj  on  A. 

For  the  linear  part  of  the  objective  function  in  problem  (1.1),  Px^, 
obviously  we  have 


P(x£  +  d£)  =  Px£  +  Pd^  ,  Vxg,  Vd2 


F  ( x  +  d)  =  F(x)  +  VF(x)d  +  o(  lldj  N ) . 

And  for  the  terms  related  to  G,.(x  +  d)  in  exact  penalty  function  P(x  +  d),  we 
have  similar  results  and  so  we  get  (4.4). 
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Lemma  4.3  If  the  functions  f  and  g.  are  continuously  differentiable,  then  for 
any  x  «ft'\  ft*  ,  any  Pj  <  1,  there  must  exist  a  »  a(x)  >  0  and  e  *  e(x)  >  0,  such 

A  A  A  % 

that  when  x€N(x,e)nft'  and  0  <  a  <  a  ,  the  optimal  solution  d  ■  d(a,x)  of  the 
subproblem 


min  R(x,d) 
s.t.  Ildjll  <  a 
ld2ll  <  M 
A(x  +  d)  <  C 


(4.5) 


must  satisfy  the  inequality 

0(a,x)  .  !S*1  :  K»  ♦  3)  >  Pl 

p(x)  -  R(x,d) 

A  A  A  A 

where  N(x,e)  A  (x  :  llx  -  xll  <  e) 


(4.6) 


Proof 

/v 

Since  x  4  ft*,  that  means 

OdD(i).  (4.7) 

A  A 

Because  3P(x)  is  a  compact  convex  set  and  {ZA^a. |1€l'(x),  A^  >  0}  is  a  finitely 
generated  convex  cone,  and  it  is  a  closed  cone  ([15],  Th.  19.1).  As  the  sum 

A 

set  of  these  two  sets  we  know  D(x)  is  also  a  closed  convex  set  [17].  According 
to  the  separation  theorem  for  convex  sets,  (4.7)  implies  that  there  exists  a 

A 

vector  t,  II t II  *  1,  and  a  number  x  >  0,  such  that  for  all  s  €  D(x) 


st  <  -t  <  0  (4.8) 


Especially  for  Vs  6  3P(x)  c  D(x)  the  above  inequality  holds.  So  we  have 


max  st  <  -t  <  0 
s«3p(x) 


8 


By  virtue  of  (2.10),  the  above  Inequality  means 

p“(x,t)  ■  H"  |“P  f(x.+  at>  -  pt*)  <  -T 

a+0  0 

A 

Hence  there  exist  positive  numbers  7  and  a  ,  such  that  when  x  e  N(x,7),  0  <  a  <  a  , 
we  have 

P(x  ♦  at)  -  P(x)  <  _t  (49) 


On  the  other  hand,  (4.8)  also  implies  that 


a^  <  0  ,  Viel'(x) 


(4.10) 


Otherwise,  if  a^t  >  0  for  some  1el'(x),  then  taking  s'  ■  A^  +  deD(x)  (de3P(x)) 
and  letting  the  corresponding  A^  -*•  00  .  we  shall  get  a  result  which  contradicts 
(4.8). 

Ax 

For  every  1  i  !'(*),  if  we  set 


■6.  A  a^  -  ci  <  0 


then  there  must  exist  an  such  that  when  x  e  N(x,e^),  we  have 

A 

61 

aix-  ci  <mT 

By  selecting  >  0  such  that  when  0  <  a  < 

A 

6. 


aa^t  < 


i 


now 


A  A 

for  Vi  4  I ' (x) ,  i  f  x  e  N(x ,e^ )  and  0  <  a  <  ,  we  have 

a^(x  +  at)  -  c^  <  (a^x  -  c^)  +  oa^t  <  0 


Now  we  take 


e  A  min,,  {e,e. }  ,  a  A  min^  {a, a.} 
*  idi'(x)  1  i*r(x)  1 


(4.11) 


and  for  any  x  e  N(x,e)  n  (!',  any  0  <  a  <  a  ,  because  of  (4.10)  and  (4.11),  we 
obtain 

a1  (x  +  at)  <  ci  ,  1  **  1 . 1  (4.12) 

and  at  the  same  time.  Inequality  (4.9)  holds,  too. 

For  such  x  and  a  let  us  consider  the  optimal  solution  d(a,x)  and  corresponding 
optimal  value  q(x,d)  of  the  subproblem  (4.5). 

From  Lemma  4.2,  it  is  clear  that 
AP(x,d)  =  P(x)  -  P(x  +  d) 

■  P(x)  -  R(x,d)  +  oddjlt) 

*  AR(x,d)  +  o(a)  (4.13) 

(Since  HdjH  <  a  )  and  so, 

% 

a(a,x)  -  AP(x4}  *  1  +  (4.14) 

AR(x,d)  AR(x,d) 

Now  we  are  going  to  get  an  asymptotic  estimation  for  AR(x,d). 

a  A 

Let  d  A  at,  then  according  to  expressslon  (4.12),  we  know  that  d  is  a 
feasible  solution  of  problem  (4.5),  so  we  have 
R(x,d)  <  R(x,d) 
and 

AR(x,d)  >  AR(x,d)  (4.15) 

4 

By  (4.9), 

AP(x,d)  *  P(x)  -  P(x  +  d)  >  ^a 
Similarly  to  (4.13)  ,we  can  have 

AR(x,d)  *  AP(x,d)  +  o(a)  >  ~cx  +  o(a) 

A 

(Since  lid j n  »  #atjH  <  all t H  *  a  .)  Substituting  above  expression  for  the  right 
side  of  (4.15)  ,we  get 


AR(x,d)  >  |-a+  o(a) 

and  hence  the  right  side  of  (4.14)  tends  to  1  when  a  approaches  to  zero,  and 

A  A  ^ 

which  means  that  for  every  pj  <  1,  there  exists  a  *  a(x)  (<a),  such  that  when 

A  /V  A 

0  <  a  <  a  ,  for  every  x  e  N(x,e)nft‘,  we  have  (4.6). 


Lemma  4.4  If  bounded  closed  set  Rcft'N  ft*,  then  for  every  p^e(0,l),  there  must 
exist  a*  >  0,  such  that  for  any  ae(0,a*]  and  arbitrary  x  e  R,  the  quotient  a( a,x) 
determined  by  (4.6)  satisfies  the  condition: 


a(a,x)  >  pj 


(4.16) 


Proof 


According  to  Lemma  4.3,  for  every  x  e  R,  there  are  e(x)  >  0  and  a(x)  >  0 


such  that 


x  e  N(x,e(x) ) n R 

A 

0  <  a  <  a(x) 


o(a,x)  >  pj 


All  these  N(x,e(x))  cover  the  bounded  and  closed  set  R,  so  there  must  exist 

A  A  A  A 

finite  number  of  neighborhoods  N(x^,ej),  ....  N(Xj.Cj),  such  that 

RC  U  N(i.,e.). 
ixl  1  1 

/A  A 

From  the  finite  number  of  *  a(x^),  which  correspond  to  those  neighborhoods, 
we  take  their  minimum  value 


min  {oij} 
1~l,«..,j 


and  it  is  clear  that  when  0  <  a  <  a*,  (4.16)  holds  for  all  x  e  R. 


Theorem  4.5  If  the  constrained  level  set  of  exact  penalty  function  P(x)  about 
the  initial  point  x^  of  the  algorithm, 

L(x^)  A  {x  :  x  e  fi'  and  P(x)  <  P(x^)} 

is  bounded,  and  the  sequence  of  iterative  points  {xv  '}  obtained  from  SLP 

fkl 

algorithm  is  infinite,  then  {x  '}  must  have  limit  points,  and  furthermore,  at 
least  one  of  them  is  a  constrained  stationary  point  of  P(x)  In  n* . 

Proof 

1 k+1 1  fkl 

According  to  the  rule  of  the  algorithm,  we  have  P(xv  ')  <  P(xv  ),  and 

hence  {x^}  c  L(x^),  which  means  {x^}  must  have  at  least  one  limit  point 

because  of  the  boundness  of  the  level  set  L(x^). 

We  now  prove  the  second  conclusion  by  contradiction.  Suppose  none  of  the 
(k) 

limit  points  of  {xv  '}  were  constrained  stationary  points,  then  for  every 

x  e  L(x^)nn*,  there  exists  an  e  *  c(x)  >  0,  such  that  neighborhood  N(x,e) 

( k ) 

contains  no  point  of  {xv  '}.  We  denote  the  union  of  all  these  neighborhoods  as  Z: 

Z  s  N(x,e{x) ) 

xel(xvu)nn* 

then 

{x(kh  C  R  A  L(x(1^)\  Z  C  L(x(1))\  n*. 


Since  R  defined  in  this  wav  is  obviously  a  closed  bounded  set,  according  to 


(10- 


Lemma  4.4  and  the  algorithm,  if  ot^  <  a*,  then  we  must  have  a(ak,xv  ')  >  pj 


and  ak+^  will  no  longer  be  less  than  a^:  ak+^  >  ak,  and  it  means  that 


ak  >  min  {otj,—}  A  B  ( k=l  ,2 .... ) 


(4.17) 


From  (4.17)  we  know  that  there  must  exist  a  subsequence  of  {xVR;}, 

satisfying 


,00 


(4.18) 


°^aki »x^  1  '  >  P1  * 

Otherwise,  if  there  were  a  kg,  such  that  when  k  >  kg, 

0{ak*x^)  <  pi 

then  by  the  rule  of  SLP  algorithm, 

“k+l  =  *  (k  >  ^o) 

which  leads  to  0,  contradicting  (4.17). 

In  the  bounded  sequence  {x'  1/},  there  must  contain  a  convergent  subsequence 
(kl 

of  {x'  For  simplicity,  without  loss  of  generality  we  assume 

X(^  +x*  e  R  (i  -*•  ~) 

From  (4.18)  and  algorithm,  we  know  that 
x(ki+l)  =  x(ki)  +  ^(kj) 
and 

P(x(ki+1))-  P(x(kiJ)  ^  px  ARfc^.x^)  ,  i-1,2,...  (4.19) 

(k- ) 

Now  we  estimate  AR(a.  ,xv  i')«  Consider  the  following  supplementary  problem 

Ki 

min  R(x*,d) 
s.t.  ii dj n  <  | 

IldJI  <  M  -  # 

2  2  (4.20) 

A(x*  +  d)  <  C 

and  assume  the  optimal  solution  of  it  is  d*  (because  M  can  be  taken  as  large 
as  one  likes,  and  8,  if  necessary,  can  be  decreased  from  the  value  defined  in 


23 


(4.19),  we  can  always  regard  M  -  -|  as  a  positive  number). 
Theorem  4.1,  we  have 

AR(f,x*)  =  P(x*)  -  R(x*,d*)  A  e  >  0 

A 

Set  x  =  x*  +  d*,  and  in  virtue  of  continuity,  there  is  an  i 
iSi0- 

P(x(ki')-  R(x(kl',x  -  x(k'h  >  | 
and 

llx^M  .  x* II  <  | 

Now,  from  the  above  inequality  and  (4.17),  when  i  >  iQ,  we 
II (x  -  x(ki))1»  <  II (x  -  x*)1fl  +  II (x*  -  x^^Jjll 
<  6  <  ak , 

Similarly,  we  know  that 

II (x  -  x(ki))2ll  <  M 
and  clearly, 

A[x(k^  +  (x  -  x(k^)]  =  A(x*  +  d*)  <  c 
The  above  three  inequalities  indicate  that  vector 
d!ki>  a  ;  -  x^f* 

is  a  feasible  point  of  the  k-j -th  subproblem  (4.5)  (i  >  i  ). 
R(x(ki),d(ki))  <  R(x(ki)  ,  ^  -  x(ki}) 


Since  x*  i  ft*,  by 


>  0  such  that  when 


(4.21) 


have 


Hence, 


and 
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AR(a.  ,  x(kiJ)  >  P(x(kl))  -  R(x(ki)  *  x  -  x(k^) 
Ki 


>  f 


(See  (4.23)) 


combining  above  inequality  and  (4.19),  we  obtain 

P(x(ki+1))  .  p(x<ki))  >  ^  (i  >  iQ) 

which  contradicts  the  convergence  of  the  following  series  of  positive  terms: 


Z  [P(x(ki+1))  -  P(x(ki))3  ^  Z  [Ptx^i+l^  -  P(x(k^)] 
i=l  i=l 


=  P(x*)  -  P(x^kl^)  <  +<»  . 
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Since  Griffith  and  Stewart  firstly  proposed  as  successive  linear 
programming  method  for  solving  general  nonlinear  programming  problems, 
such  methods  have  been  widely  used  in  practice  because  of  their  ease  of 
implementation  and  their  ability  to  deal  with  large  scale  problems. 

However,  neither  the  original  version,  nor  a  more  recent  one  contain 
convergence  proofs  possibly  because  of  non-robustness  of  their  algorithms. 

Using  exact  penalty  functions  and  Levenberg-Marquardtlike  steps,  an  improved 
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20.  ABSTRACT  (continued) 

algorithm  has  been  recently  devised.  In  this  paper,  we  give  this  modified 
SLP  method  a  theoretical  analysis  and  convergence  proof,  and  thereby 
provide  a  sound  basis  for  It. 
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